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Collaborative Design of Educational Indicator Systems 
in Developing Countries: 
An Interim Report on an lEES Project Initiative 

I. INTRODUCTION 

The design of educational indicator systems is presently a "hot topic" in both the 
industrial and developing worlds (Bottani & Delfau, 1990; Chapman, 1990; Odden, 1990; 
Windham, 1988a, 1988b) as better use of existing data at all levels holds out the hope of 
rationalizing the macro-management of educational systems and directing attention to the 
real impediments to improved performance and greater equity. 

lEES Project activities in this domain were bom of some related concerns, though 
ones that changed form significantly in the course of implementation. Evaluation of the 
first five years of lEES Project activity suggested a number of positive effeccs, but also a 
need to better document project impacts and the evolution of the baseline situation and 
socio-economic environment of the educational systems in question. Project staff therefore 
decided to begin work on a prototype indicator system for tracking the overall state of the 
education systems in participating countries, in part as an ongoing and greatly simplified 
form of sector assessment. 

Once the team responsible for this effort began to bring together current research and 
thinking on indicator systems with its own experience in educational data use in developing 
countries, it quickly became clear that an additional two dimensions would be needed: first, 
incorporation into the model of qualitative data on educational processes, habitually the 
weak point of indicator systems; and, second, major and structural provision for host 
country counterpart participation in indicator system design and data definition, as well as 
anticipation of considerable variance and country specificity in design characteristics. 

In the rest of this paper, we briefly describe our conceptual analysis cf educational 
indicator systems, the prototype model of primary education developed, and the process 
envisaged (and now underway) for host country elaboration and modification of the model. 
We will also discuss the potential for utilizing the indicator system both for in-country 
educational planning and staff training purposes. 

II. CONCEPTUAL ANALYSIS OF EDUCATIONAL INDICATOR SYSTEMS 

A* The Nature of Indicators and Indicator Systems 

Recently there has appeared an increasing amount of literature about indicators for 
educational systems, provoked by concern with educational accountability in this country 
and spearheaded by the OECD Indicators Project in the rest of the industrialized world, as 
well as by donors' attempts to improve educational management and soften the blow of 
diminished educational resources in less-developed countries (LDCs). 

There is no uniform definition of the notion of an "indicator,' however. Chapman 
(1990, p. 229) terms indicators "proxies used to represent the underlying reality of a system 
or program" and notes that they are necessarily an "oversimplification" of this reality. 
Cobbe (1989, p. 3) makes a distinction in his monograph between raw data, statistics and 



indicators, noting that the latter "should permit immediate - or nearly immediate - 
inferences about the performance of the [educational] system from the point of view of the 
objectives of the system." 

Oakes (1986) defines an educational indicator as "a statistic about the educational 
system that reveals something about its performance or health;" and Smith (1988) funher 
suggests that it "either assesses or is related to a desired outcome of the education system or 
describes a core feature of that system." Odden (1990, p. 24) points out that an "indicator 
system" is considerably more than a single indicator or statistic. It should 

provide measures of various components of the educational system as well as in- 
formation about how those components work together to produce the condition 
of the system and changes in the condition of the system over time. 

Most researchers identify indicators with composite indices of the operations of the 
educational system designed to reveal some critical aspect of its operation. Johnstone 
(1984) goes so far as to restrict the term to rather complex compounds of data providing a 
highly synthetic picture of the system. 

Such an approach seems unnecessarily restrictive, however. The essential notion is 
simply that of particularly meaningful data that point to and describe a significant 
characteristic of the educational system. In some cases, a single type of data may do this 
(Nuttall [1990] calls these "primary indicators"); in other cases, more processed composites 
of raw data or statistics may be used ("secondary indicators," in Nuttall's pariance). In all 
cases, commentators tend to emphasize the fact that, as Bottani (1990, p. 337) puts it, 
"indicators do not explain; they only point." Some additional resources must be called upon 
in order to interpret the data that an indicator system produces. 

B. What is Worth Indicating? 

If indicator systems only point, what are we pointing at? From a generic and "rational" 
point of view, the underlying concern in indicator system development seems to lie with 
discovering whether and how - or "to what degree" and "in what way" - the educational 
system is accomplishing its goals. Since those goals will generally be quite country-specific, 
and given that there is often considerable debate within a country as to what those goals 
ought to be, it follows that indicator systems may also need to be quite idiosyncratic and 
subject to modification, a tendency tempered only by some measure of concern with 
consistency and comparison over space and time. 

At the same time, it is evident that the lEES Project, like other multinational 
endeavors of donor agencies, is predicated on the notion that there are some universal 
criteria of educational system performance, related in particular to issues of efficiency and 
quality. How universal are these criteria and how much uniformity can or should they lend 
to a multinational indicators project? 

Educational Efficiency: Efficiency in the delivery of educational services is a central 
concern of the lEES Project. It is the Project's "middle name." A good deal of conceptual 
work has been done on the topic under the aegis of the project, particularly the substantial 
monographs by Windham (1988a, 1988b). Much of that analysis is implicitly adopted here. 



At the same time, we feel constrained to note that injunctions to maximize the ratio of! 
educational outputs or outcomes to educational inputs frequently glaze over three 
fundamental issues: 

First, technically speaking, the concept of efficiency is only meaningful if you correctly 
specify (and measure) the outputs or outcomes in question. Desired educational outputs 
and outcomes are not simple, and they may vary significantly from one country or region to 
another. They may involve affective as well as cognitive results, group as well as individual 
effects, and distributional as well as simimational considerations. Policy makers may in 
reality be trying to maximize an entire vector or set of outcomes - Windham refers to it as 
the outcome "mix", - including (conceivably) the desideratum that the relative position of 
one group improve relative to that of another. Issues like these are often circumvented by 
imposing at the outset a conventional series of objectives and desired outcomes - generally 
those in vogue with major donors. 

Second, efficiency is a ratio, not an absolute magnitude. Applying the perfect 
competition and general equilibrium postulates of neoclassical economics, it is generally 
assumed that efficient methods that are applicable at one resource level are generalizable 
across others as well. This may not be true; and, if not, the reasoning is in serious trouble. 
Also, for related reasons, as Cobbe (1989) points out, an efficiency criterion can be 
substantively ambiguous, particularly if a short-term view is adopted. For example, cutting 
inputs by one-half might "mathematically" improve efficiency in short-run perspective, yet 
have ruinous longer-term effects. 

Third, educational systems need to be efficient both in generating AND in expending 
resources, and these two objectives are not always perfectly compatible. A system which 
produces less outputs per unit of input but manages to generate or elicit more input - e.g., 
community or private financing - may be better adapted to accomplishing national 
objectives than a more "efficient" one. In a sense, this point mirrors the economic 
distinction between technical and allocative efficiency, since the latter takes account of the 
relative prices of inputs and outputs and therefore the valuation that (public or private) 
consumers place on educational system performance. 

Overall, Windham suggests that "the efficiency concept is a neutral device" (1988, p. 9), 
but that only holds under one of two very restrictive conditions: (1) if the term is given no 
content, in which case it is operationally useless; or (2) if there is substantial agreement 
among all stakeholders concerning the objectives or the "outputs" that are being maximized. 
To make the concept operational, some set of objectives or desired outputs must be 
specified; and the question then immediately becomes "Whose objectives? Desired by 
whom?" 

Educational Quality: Fuller (1988) chronicles four different phases in the definition of 
educational "quality" - from an initial stress on the mix of material outputs, through a 
period of increased emphasis on the importance of individual abilities and perceptions, to a 
phase of growing awareness of the centrality of cultural and social patterns that shape 
school process, and on to more recent awareness of the non-school-related and political 
objectives that "educational quality" campaigns may fulfill. Cobbe (1988, p. 3) identifies 



educational quality with the "value added" by the system, though this makes it nearly 
synonymous with the efficiency concepts discussed above. 

Easton and Cayhana's paper on the efficiency and quality of technical education in 
Indonesia (Easton & Cayhana, 1989, pp. 3-4) notes two main tendencies in efforts to 
operationalize the notion of educational quality: 

On the one hand, quality is defined as embodiment or approximation of charac- 
teristics that are socially accepted as proof of excellence. Thus, if all teachers in 
an academic secondary school have Master's degrees, the group will be con- 
sidered a high quality staff. One the other hand, quality is defined as proven 
ability to produce results - in short, the argument that 'a tree is known by its 
fruits.' 

Moreover, "educational quality ' in the former sense is very often invoked in 
evaluations and policy studies for two reasons worth noting: first, as a substitute or proxy for 
efficiency measures, because genuine output data are so frequently unavailable; and 
second, because this open-ended concept is felt to capture more of the multiple and 
sometimes unquantifiable dimensions of educational outcomes than stricter input-output 
measures. Concern with educational quality therefore opens the door to the idea that other 
than quantitative data may need to be incorporated into an indicator system - or used in 
tandem with it - in order adequately to describe and analyze the current state of the 
educational system. 

C. Specifying the Underlying Conceptual Model 

Two points are made repeatedly in current literature on educational indicators: first, 
that indicators "do not explain, they simply point;" and second - partly as a consequence - 
that one needs to take some care to think out the underlying model of educational process 
on which the indicators are based, and which will serve as a basis for interpreting them. 
The model need not be very complex. The Rand Corporation scheme presented in Odden 
(1990) includes only three types of inputs, four types of processes and three categories of 
outputs. But most authors maintain that some such model is implicit in any set of indicators 
anyway, and that it is therefore best to get the subject out in the open and examine it. 

We concur with the need to give some careful thought to the model of educational 
processes that underlies an indicator system. Most lEES studies and documents employ 
some form of a "CIPP" framework (in fact "CIPOO" insofar as the "product" is differentiated 
into "output" and "outcome"). The CIPP framework is not really a model in the sense meant 
above, however, but simply a set of categories (based albeit on a production function 
analogy) for thinking about educational processes. The question remains, "What kinds of 
'inputs,' 'processes,' 'outputs' and 'outcomes' will be put in the model?" and "Who will 
make the decision?" 

D. Choosing Indicators: Criteria and Methods 

The next step is choosing the specific indicators to be tracked. We feel that the choice 
should be made in accordance with both conceptual and practical criteria. Before 
discussing those criteria, however, there is a prior question concerning the "population" of 



possible indicators from which the choice will be made. In order to avoid overly or 
artificially constraining that initial set (and therefore all the choices made from it), we 
decided to inventory a large number of potential indicators, cross-classifying them by 
conceptual category or cluster. The choice criteria were then to be applied to this stratified 
grouping. 

Chapman (1990, p. 229) suggests that "the appropriateness-of particular indicators is 
judged in terms of their fidelity with the underlying reality they represent, the extent to 
which the indicators are relevant and understandable by the data users, and the extent to 
which data on these indicators can be collected and analyzed in a cost-effective manner." 
We attempt to include these criteria and perspectives in the discussion below. 

1. Conceptual Criteria 

Conceptual criteria essentially involve considerations similar to the ones governing the 
articulation of the underlying model. Since only a relatively few indicators will be 
systematically monitored, one wants to be sure that they relate to "key" aspects of the 
educational system. What does "key" mean, however, and who decides which indicators 
meet this criterion? 

Considering the first question first - we think that the word "key" essentially means 
that the aspects of the educational system in question are judged (by appropriate parties 
and in an appropriate manner) to be particularly expressive of its current state or level of 
quahty, or particularly critical to its performance in achieving (their) priority objectives. 
Two technical issues are at stake here: the first concerns validity, and the second has to do 
with causality. 

The validity question involves the degree to which the particular indicators chosen 
validly represent the underlying characteristic of the educational system that one wishes to 
measure, an argument that can be made on either conceptual (construct validity) or 
empirical (concurrent validity) grounds. The causality issue concerns the degree to which 
these underlying characteristics are in fact related in some cause-and-effect manner to 
achievement of the given system performance objectives. 

On both grounds, we can use - though advisedly - some of the literature reviewed by 
Fuller (1986, 1987) and others that seeks to identify key factors in improving educational 
quality in LDC schools. Obviously research on the countries in question has the greatest 
face validity, but is frequently hard to come by. 

2. Practical Criteria 

Practical criteria to use in choosing indicators are essentially of two kinds: (1) data 
availability considerations, and (2) data quality considerations. They will be broached in 
that order. 

Data availability considerations are relatively straightforward, but are nonetheless 
worth careful examination. From the point of view of a centrally-administered 
international project, one could establish a hierarchy from "more to less available" with 
gradations something like the following: 



(a) Available in easily accessed international publications. 

(b) Available in more remote international publications. 

(c) Available in easily-accessed international data bases. 

(d) Available in national publications easily accessed in-country. 

(e) Available in national publications requiring specially-authorized access in-country. 

(f) Available in international data bases requiring special search or access. 

(g) Available in national data bases requiring special search or access in-country. 

(h) Available in-country in dispersed or raw data form requiring considerable 
collection or collation effort. 

The order and relevance of this scale is obviously different if one is situated at the 
national or regional level, and appropriate changes should be made. In any case, there is 
some implicit rank order of availability characterizing the different forms of data that one 
might want to include in an indicator system. 

Note that the time period must also be specified in determining availability. Data for a 
given indicator may be fairly accessible for the 1988-90 time period, for example, but much 
more difficult to obtain for the 1984-87 period. 

Data quality: This is both the most critical and the touchiest practical criterion, it 
seems to us, a problem of major dimensions that is nonetheless carefully skirted in most 
discussions of educational indicators or management information systems in LDCs. The 
issue can be politically sensitive, but it is also absolutely central to the development of any 
tracking system. The articles of Chapman (1990) and Chapman & Boyd (1988) are among 
the very few writings we have found to treat the topic straight-ou, though the authors limit 
themselves to a particular technical treatment of the issue. 

The underlying question concerns the margin of error in data; and the basic principle 
to assert, we feel, is that all information contains elements of error, and any responsible 
presentation of data should include an estimate of this margin or at least a discussion of the 
sources of error. The underlying types of error are threefold, and can moreover compound 
each other: reUability, validity, and sampling (representativity). Validity errors have 
already been discussed. We briefly consider the other two types in the following paragraphs. 

Reliability: Missing information, fanciful estimates, miscopying, falsifications, 
misunderstanding of instruments, and a legion of other miscues increase the margin of 
error in the data (or, technically speaking, increase the variation that would be found in the 
results of successive efforts to determine the same information by separate means). It is 
important to get some handle on this issue. 

In a certain sense, the problem may be compounded by the use of composite 
indicators, (e.g., if our estimate of enrollments has a margin of error of +/- 10% and our 
information on numbers of classrooms has a +/- 20% confidence interval, the range for the 
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pupil/classroom ratio computed from this data could vary by as much as +/- 29% around 
the true value.) 

Representathity or sampling error: What population do the data represent with 
reasonable accuracy? Often data presented as national in scope have only been collected 
on a distinctly non-random subset of locations or units, perhaps because of poor response 
rates. This situation does not necessarily invalidate the information in and of itself. Decent 
data on a particular subset (which is usually representative of something, even if not of the 
whole country) is immensely better than no data, and may be very informative, provided 
that one takes account of its limitations. 

At first glance, it might seem impossible to get information on - and take account of - 
these sources and margins of error. Given that the whole topic has potential for 
undermining the credibility of any information system, one could then understand the 
inclination to scrupulously ignore it. But the main point is simply to refrain from throwing 
the baby out with the bath water; or, as the French say, to remember that "the best is an 
enemy of the good." The main point is to start taking some acvcount of sources and margins 
of error, to be ready to exclude the data that seems most suspect in this regard, and to begin 
pushing for inclusion of margin of error information along with any data collection or 
analysis activity. 

E. Whose Conception? 

A number of critical questions arise throughout the foregoing reflection on the 
conceptual framework and practical guidelines for devising an indicator system: namely, 
whose conceptual framework shall underlie the indicator system? how shall it be devised? 
and who shall specify the criteria for choosing indicators and the complementary roles of 
qualitative and quantitative inputs? Though there is incontestably an international 
language of educational planning and research currently largely dominated by economics, 
there are a number of reasons to assert that actual indicator systems may and should vary 
significantly by locality. 

The first reason is that, as noted above, efficiency has little meaning until 
operationalized by the specification of objectives, and those largely depend on the 
development strategy and perspectives of the countries in question. The second is that 
actual data availability may vary enormously as a function of other conditions within the 
host country environment. A third is that the ultimate utility of an indicator system 
depends entirely on one's ability to interpret the data, and most indicator systems therefore 
require a complementary set of qualitative insights that are normally highly country-specific. 

To the degree that there is effective host country participation, therefore, the potential 
for variation in the indicators from one country to another will significantly increase, even 
though there certainly are international paradigms of educational development that exert 
major conceptual influence nearly everywhere. Moreover, there is also a potential here for 
variation over time. The notion of a key or critical aspect of an educational system has a 
time-bound quality and is strategy-related - aspects that are thought to be critical at one 
poiiit in time may not be at another period. The high value (and rare occurrence) of 
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longitudinal analysis should drive proponents of the system to resist too many changes, but 
the likelihood of variation in data specification over time must be taken into account. 

The conceptual analysis in and of itself therefore drives us tO emphasize host-country 
participation and local "ownership" of the indicator system design process. Of course, this 
shift in center of gravity does not automatically answer and resolve the "cui bono?" or "who 
decides?" question. The same issues are simply transposed to a-host-country level, and one 
must ask again whose criteria and conception are being used and to what degree they match 
those of the supposed beneficiaries or ultimate underwriters of the educational system. We 
will return to this topic a little further on in discussing field implementation strategy. 

III. DEVELOPMENT OF A PROTOTYPE MODEL 

A* Steps in the Process 

Our original intention was to move from consideration of these conceptual bases to 
the development of a simple indicator model applicable to all lEES countries and thence to 
collection and analysis of data that would illuminate trends in system performance and set a 
framework for discussion of Project impacts. As just explained, however, we came out of 
the original conceptual stage with a different set of proposed orientations. At that point, 
the ideal procedure would seem to have been an. interactive design process including 
substantial host-country participation from the outset. For logistic and budgetary reasons, 
however, this was not feasible. Our fallback strategy was to proceed with design of an initial 
prototype model, using our collective field experience - and that of the rest of lEES staff - 
as a sounding board, then to treat the result as a stimulus for country-specific work and to 
develop a strategy for recentering the debate in host-country locations. In this section of 
the paper, we briefly describe the model development process and its initial results. In the 
next section, we turn to the question of the process for recentering the debate in 
participating countries. 

The model elaboration process included the following steps: First, review of the 
theoretical bases of indicator systems and development of an initial conceptual framework. 
Second, inventory of potential indicators of system performance and classification into the 
categories of the conceptual framework. Third, choice of a set of indicators to compose the 
initial "indicator system." Fourth, submission of this framework to lEES staff and other 
colleagues for critique, discussion and modification. 

This work was undertaken in the course of the 1990-1991 academic year. Candidate 
indicators were drawn from Windham ^ monographs (1988a, 1988b), sector assessment 
documents. World Bank and UNESCO publications, the writings of Fuller (e.g., 1986, 
1987) and others on educational quality concerns in developing countries, and experience 
with indicator systems in Indonesia, Haiti and Botswana. To simplify the conceptual task, 
we based all reflection on the example of a country's primary education system- 

B. Initial Conceptual Model 

The underlying conceptual model that we adopted was based to a certain extent on 
work previously done in Haiti and consists of three tiers: descriptive indicators, efficiency 
indicators, and equity indicators. At the descriptive level - portrayed on the adjoining 



table - the characteristics of the educational subsystem in question are organized into fif- 
teen domains in roughly production-function fashion: beginning, that is, with context and 
proceeding through inputs, processes, outputs and outcomes of education. In an effort at 
simplification, we decided to choose for the initial prototype model two summary indicators 
per descriptive domain - or thirty in all - based on criteria of likely availability and 
relevance to overall issues of educational quality and efficiency.^ It subsequently proved dif- 
ficult to narrow choices down this far without reference to the particular countries where 
the work would be done, so we retained an average of four proposed indicators per domain, 
counting on subsequent in-country work for refinement and pruning. The actual indicators 
and the rationale are laid out in Appendix A. 

The next tier of the model concerns efficiency indicators. Strictly speaking, efficiency 
is a ratio of outputs to inputs. An efficiency measure is therefore generally a secondary 
indicator which is based on two or more primary indicators and may span across domains of 
a model like the one presented above for the primary education subsystem. 

Each of the domains in the model may of course have its own internal efficiency 
measures, involving its own particular inputs and outputs. For example, the input to the 
school management domain may be dollars, and the output a certain number of services 
provided to teachers and students by school administration. These two measures could 
then be combined into a "local" efficiency indicator concerning purely the realm of school 
management. For the most part, however, we are dealing with more global composite 
measures. 

Some of the descriptive indicators included in the list above are already implicit 
efficiency measures. Any form of completion or graduation rate, for example, is of this 
nature, since the "rate" compares those who succeeded (output) to all those who took part 
(input). We propose in addition to track three kinds of composite efficiency indicators 
based on the descriptive indicators already detailed: 

1) Rnrollment ratios 

Essentially total enrollment over the size of the appropriate age cohort in the 
population. 

(2) Resource sufficiency r - 

These include textbooks per student and non-salary expenditures per student. 

(3) Resource cost per graduate 

Actual costs per graduate are too complicated too compute and require cost 
studies to which we will probably not have access. A simpler measure is 
graduates per teacher-year. 

The third tier concerns equity indicators. Establishing equity indicators essentially 
involves attempting to disaggregate the types of data already proposed for assessment of the 
overall primary school system. To keep from exponentially mcreasing the amount of work 
to be done, the number of equity indicators, and the number of axes of disaggregation. 



should be kept modest. We suggest three axes of disaggregation and four indicators for 
equity comparisons: 

Axes, of disaggregation 

Significant geographic or administrative regions 
Private/public education 
Gender (male/female) 

Equity indicators 
Net enrollment ratio 
Cycle completion rate 
Textbooks/student 
Teacher training 



C. Debate and Modification 

Discussion of this overall model with lEES Project staff proved to be a highly 
instructive exercise and yielded a number of suggestions for improvement. Notable among 
these was the recommendation that at both the central and the country levels a distinction 
be made between an "ideal" version and a "feasible" version of the model. 

The underlying idea is to identify and distinguish both a "best case" and a "likely" set of 
indicators. The first one gives all participants an occasion and an incentive to think about 
the kinds of information that they would really like to have in order to inform educational 
decisions bearing on system efficiency. It also gives us some protection against the danger 
of getting mired down in "least common denominator" approaches that lead to collection of 
an incoherent assemblage of data with borderline validity simply because it was (all that 
was) available. 

The second approach (development of "feasible" indicator sets) provides a reality 
check against impractical or inappropriate model specifications born of too much concern 
with conceptual elegance or completeness and insufficient attention to what is currently 
possible in different country settings. We anticipate that iterative review of the ideal and 
feasible models will result in (a) an operational (feasible) model that is conceptually 
tighter, and (b) a better articulated optimal model that will be increasingly useful for 
purposes of training, policy discussion and educational planning. 

IV. IMPLEMENTATION IN THE FIELD: PROCESSES ENVISAGED 

A. Development of country-specific models 

The next step foreseen in the process is to move the center of gravity in the 
development of the ideal and feasible indicators sets to three lEES participating countries. 
At present, there have been expressions of interest, but the final "self-selection" of these 
countries has not been made, nor has the exact process to be respected in-country been 
specified. This will doubtless vary from one location from another and be developed 
largely in situ, with the participation and support of lEES Resident Technical Assistants, 
where applicable. The minimal objective of the exercise will be to use the proposed system 
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simply as a tool for discussion and staff training, and to enlist the help of host country 
counterparts in modifying and improving a model that the lEES Project will then use to 
track trends in educational system performance in the coxmtry. 

If there is interest in participating countries, a good deal more can be done with the 
resulting system, or with the process of developing it The exercise can serve as a basis for 
reviewing current uses of educational management information systems, designing means 
for interpreting their outputs, integrating qualitative information into the interpretation 
process, specifying types of local process research to be targeted for support, and/or training 
staff at different levels in the related analytic tasks. 

B. Related Issues 

Four major issues that merit exploration in tandem with this process are worth 
mentioning here. The first concerns the complementary roles of qualitative and 
quantitative outputs in educational decision-making, and the best means for ensuring a 
supply of relevant qualitative insight. Process data is the habitual weak point of indicator 
systems. We therefore envisage, as one component activity, a careful census of the 
qualitative and process studies currently underway in the countries in question (and 
potentially the encouragement of others through the lEES research agenda) in order to fill 
the gap and lay a firmer foundation for analysis. 

A second and closely related issue concerns investigating actual patterns of use of data 
in educational agencies and pinpointing factors that determine their utility. The "sociology" 
and politics of data use is a topic as centrally important to EMIS operations as it is sensitive, 
and certainly merits thoughtful recognition. 

A third important topic is related to the appropriate mix of nationwide and sample 
data in an indicator system. A number of more refined quantitative indicators - as well as 
most types of qualitative information - cannot be reliably collected for the whole country 
but can be very usefully approximated through focused sample studies. Developing this 
methodology as part of an EMIS merits closer attention. 

A fourth critical subject concerns the relationship to be developed between national 
management information and indicator systems on the one hand, and regional or local ones 
on the other. Our experience clearly suggests that overly-centralized systems where field 
staff collect data solely to feed central files or computers end up receiving increasingly 
unreliable information. It is important to develop at the same time methods and 
procedures for synthesis and use of data at local and regional levels, not only as a support 
for better decision-making at this level, but also to help ensure that field staff see the use of 
the data collection effort and remain conscientious about it. 

C. Analysis Strategy 

Analysis is another congenital weak point of indicator systems, as participants in the 
current debate about educational indicators in this coimtry have been quick to point out. 
Exactly how is one supposed to use the data gathered? Here lies one clear advantage of 
trying to articulate an underlying conceptual model, as we have begun to do above. That 
model can serve as an important aid in the analysis and interpretation of the indicator data. 
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Generally, we see tvvo related directions for analysis: (1) examination of trends over 
time; and (2) diagnostic interpretation of system status data, 

(1) Examination of trends over time. 

Part of the purpose of the indicator system is to provide a means of tracking the 
evolution of the educational system (most fundamentally vis-a-vis lEES activities) and 
spotting changes that provide early warning of serious problems or early evidence of major 
opportunities for school improvement. Consequently, a first level of analysis consists 
essentially of deducing direction and magnitude of changes in the key indicators from the 
longitudinal data assembled 

This is, of course, not quite as simple as it sounds, even assuming that enough 
sequential data on the same indicators are assembled to document a trend One problem 
that typically confounds such an effort must be resolved before analysis can begin: the 
non-comparability of data on the same indicator at different points in time. It is quite 
conceivable that over time the indicator may be defined differently, the population on 
which the data were collected may be different and/or the reliability of the methods used 
may vary markedly. An initial effort to take account of these irregularities will need to be 
made. 

(2) Interpretation and diagnosis 

In a sense, the analytical model pictured above already constitutes a framework for 
interpretation of the data and even diagnosis of the problems observed, since the network 
pictures a series of supposed causal linkages among educational system variables. If an 
anomaly or problem is observed at one level, we therefore have at least a suggestion of 
where to look for causes or contributing factors. 

Of course, what constitutes an "anomaly" or "problem" for highlighting and 
investigation can only be determined with reference to the particular objectives and norms 
of the educational system under examination. This once again supposes the active 
participation and involvement of host country planners. 

These remarks suggest the following order of action for interpretation of the data: 
First, determine trends. Second, isolate trends or persisting conditions that seem 
problematic, given the country's educational objectives (or, in their absence, accepted 
international norms.) Next, by comparative trend analysis, formulate questions and 
proposals about possible contributing conditions. Fourth, discuss the entire matter with 
host country researchers/planners and revise the analysis on this basis. Fifth, revise the 
indicator scheme and collection plan as warranted by the results of the analysis. 

D. Dissemination Phase 

The efforts described in this paper seem to us potentially to have some larger 
implications for data collection practices in education at the level of each country, as well as 
internationally. As much should be expected, since the lEES Project is in part an R&D 
endeavor with the explicit charge from AID of developing new procedures for educational 
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planning and research that may be worthy of broader dissemination. If so, how should such 
dissemination be handled? 

We think that the first critical dimension of the dissemination task lies within each 
host country. We will therefore be verifying in each case what are the existing in-country 
media for dissemination of educational research and reflection, and how we might 
collaborate with counterparts in using - and simultaneously in strengthening these media. 
A second dimension evidently concerns exchange among participating countries and/or 
with educational personnel from potentially interested nations in the same region as the 
host country. A third and final dimension embraces international dissemination and 
publication within the United States. 

In addition, we envisage publication at one or more of the above levels of a series of 
monographs to ensure broader availability of documentation on the development and 
products of this activity. Each monograph will represent a phase or particular theme of the 
project. Four distinct monographs and topics are presently anticipated: 

( 1) Overview monograph on conception, development and adoption of ap- 
propriate indicator systems for tracking educational efficiency; 

(2) Survey and analysis of research now completed or presently being conducted 
in participating countries on educational processes and quality dimensions, 
the habitual weak areas in indicator systems; 

(3) Monograph on problems of dovetailing national and local indicator systems 
and MIS's and on the staff and teacher training issues involved; and 

(4) Monograph on questions of cross-national comparison of procedures and 
resulting data from efficiency indicators systems. 

V. CONCLUSIONS 

Several limitations of the analysis and processes presented in this monograph should 
be kept in mind. The first and foremost is that the schema presented is purely and simply a 
framework for thinking about indicator systems and so a potential starting point for the 
more important phases of related work that must go in the field. A second limitation is that 
the discussion is largely theoretical. In fact, much of what needs to be done in improving 
data utilization in the field is more particular and concrete and has to do with questions of 
how useful insights are drawn from existing data, how one diagnoses the small shifts in the 
type of information collected that could have major impacts on its usefulness, how disparate 
data collection systems in a single country or region are coordinated to eliminate waste 
effort and contradictions, and so forth. The framework suggested in this paper may 
nonetheless have some value for stimulating discussion and focusing attention on some 
"common denominator" issues in indicator system management. 

In sunmiary, one could say that "a funny thing happened on the way to establishing a 
project-wide indicator system for tracking efficiency trends": we remembered the 
importance of place and process and came up instead with a plan for engaging host-country 
participants in dialogue about - and collaborative design of - more flexible and 
location-specific systems. That this should happen in the lEES Project is not surprising, for 
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the Project's finest hours and best accomplishments have lain in the impulse to translate the 
current technique and jargon of educational planning into participatory exercises, 
accessible to host country counterparts and subject to their modification, criticism and 
influence. Sector assessment as practiced in the project, for instance, for all its sometimes 
heavy jargon, positivistic assumptions and economic bias, has been widely appreciated as a 
means of making the process of foreign aid allocation and educational policy analysis more 
visible and of opening the door to greater direction of the process by host country nationals 
(Easton, 1988), A number of project activities have had, intentionally or not, major 
institutional development consequences; and most have been characterized by a concern to 
adapt research agenda to capacity-building priorities. The reorientation of the efficiency 
trends activity is thus in a sense simply a move to make it more consistent with the spirit of 
the lEES Project, 
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APPENDIX A 



DEFINITION OF MODEL DOMAINS 
WITH PROPOSED INDICATORS 

L CONTEXT 

1. SociO'Economic and Demographic Background 

This domain serves to establish the baseline context for assessing schooling in- 
puts, processes, outputs, and outcomes. Five indicators have been selected to 
represent this domain: a) total population, b) life expectancy at birth, 
c) Purchasing Power Parity GDP per capita, d) newspaper circulation per 
1000 population (to reflect literacy/communications), and e) the number of 
adults completing primary education. 

II. INPUT 

2. School Endowment 

The domain of school endowment is intended to reflect the basic physical and 
material resources available at the school site. The three indicators selected to 
reflect the availability of instructional resources are a) class to classroom ratio, 
b) whether furniture is provided, and c) the availability of instructional materials. 

3. Locall Community Support 

The degree to which local schooling is supported by the community is reflected by 
this domain. The indicators of this support are a) whether a functional PTA exists, 
b) the nature of school-community contacts, and c) the percentage of school ex- 
penditures met through non-central government sources. 

4. Central/ Regional Support 

The support offered to local schooling through the central/regional offices of the 
Ministry of Education, or other institutional provider is represented by a) the 
number of textbooks produced, b) the number of school inspection visits by MOE 
officers, and c) the overall and non-personnel Govemment/MOE expenditures. 

5. Student Characteristics 

Student population characteristics, i.e., what they bring to the schooling 
enterprise, are represented by a) total enrollment, b) the percentage of female 
students, and c) data on their health and nutritional status. This domain of stu- 
dent characteristics is distinct from how students engage in the instructional 
process, which is represented below in "student participation." 
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6. Teacher Characteristics 

The characteristics of the teacher population (status of the profession and what 
they bring to the instructional process) are represented by a) the number of 
teachers, b) the percentage of female teachers, c) training (by level), d) average 
salaries, and e) attrition rates. 

IIL PROCESS 

7. School Management 

School management refers primarily to the organization and provision of instruc- 
tional time and resources, with additional indicators of administrator qualification 
and efficiency. To represent instructional time and resources, the following in- 
dicators have been selected: a) number of school days per year, 
b) hours of instruction per day, and c) school expenditures on facilities and main- 
tenance. Indicators of administrator characteristics are d) the percentage 
of headmasters completing headmaster training, and e) a summary index of ad- 
ministrative efficiency.* 

8. Curriculum Quality 

The quality of the curriculum will be represented by summary indices of a) the de- 
gree to which the national curriculum is followed, and b) the appropriateness of 
the national curriculum.* 

9. Teacher Quality 

The quality of teaching is represented by a) the amount of teacher's time spent on 
instruction, b) the use of instructional materials for instruction, and c) the fre- 
quency of testing and feedback to students. 

10. Student Participation 

This domain refers to effective school attendance and participation in organized 
learning activities. The underlying variables concern quantitative and qualitative 
appreciation of "time on task" that is, time spent on the activities designed or in- 
tended to promote achievement of the school's learning outcomes. The indicators 
that we have chosen in this realm are a) student absenteeism rate, and b) student 
time on-task. 

IV. OUTPUT 

11. Student Attainment and Achievement 

This domain concerns the degree of achievement of primary school learning ob- 
jectives in the cognitive and psychombtor realms and includes quantitative and 

* Note: These "indicators' are a conceptual placemark, representing, to date, categories rather than actual 
dicators. 
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qualitative assessments of those scholastic "outputs." The indicators that we have 
chosen in this area are a) percentage of students reaching the last grade of the 
cycle, b) results of mastery exams, and c) student grade-to-grade progression 
rates. 

12. Student Attitudes and Aspirations 

This domain concerns the degree of acquisition of selected attitudes, or the de- 
gree of achievement of specified affective learning outcomes, by primary school 
students. Chosen indicators are a) percent of students taking admissions exam for 
next cycle and b) the general area of student occupational choices (specific in- 
dicator yet to be designated). 

OUTCOMES 

13. Later Academic Outcomes 

"Later academic outcomes" refers to the degree of success of primary school 
graduates in their further schooling, including admissions to higher level institu- 
tions of learning and completion of these subsequent cycles. The specific 
indicators chosen in this area are a) percent of graduates of the primary cycle 
entering the following cycle, and b) the number of graduates in scientific/techni- 
cal fields. 

14. Economic Outcomes 

The category "economic outcomes" refers to financial success in post-schooling 
employments and can include indicators like measures of the length of job search, 
stability of employment, and income of primary school completers or leavers. 
The specific indicators chosen are a) average earnings of graduates of cycle, and 
b) labor force status of graduates of cycle. 

15. Social and Political Outcomes 

This domain covers other dimensions of social and political behavior that may be 
influenced by primary school attendance, including social mobility, political par- 
ticipation, number of children and their schooling, etc. The indicators chosen for 
this exercise are a) the general area of political participation of graduates of cycle 
(particular indicator yet to be designated), and b) an index (yet to be specified) of 
social mobility of graduates of cycle. 

16. (Feedback to Context) 

This category simply indicates that the outcomes of primary schooling have in 
turn an impact on the characteristics of the socia economic context in which the 
school system operates. 
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